It is often desirable to have an understanding how a system performs (or ceases to perform) under a load where one or more of the system's resources is taxed in some manner. One way of doing this is to measure the number of transactions per second the system can process and/or the number of concurrent connections that can be provided for a certain period of time without exceeding a predefined target latency or returning a certain percentage of error messages. Engineers often devise various one-time load generator solutions to address their problem space at a high cost.
Conventional methods of testing a system are not ideal for a number of reasons. Various efforts of one engineer often overlap with other one-time efforts of another engineer. Further, different engineers often desire to compose tests using different languages, such as Java, C++, and Ruby, increasing the complexity of interfacing with a load generator. In addition, a machine used for testing may not have enough hardware resources (memory, processor, disk space, etc) to handle applying a load, especially if the system being tested is designed to handle large loads. By testing with a machine that does not have enough resources, test results may be skewed by the resource needs for load generation. Sometimes the main purpose of a test is to obtain results of the load itself, but other times, users may wish to know the outcome of a specific client scenario when the server is under a desired background load (in which case the load is “background noise” versus the main subject of the test). In such instances, developers may devise a multi-threaded solution where some of the threads are applying the load and some of the threads are applying the test. As another example of a problem, a test machine may be in a geographically different location from the server, and latency introduced by this distance may affect the latency results to make them unrealistic. If the user wishes to put load on a system from different internet protocol (IP) addresses (for example in a case where the requests are being throttled by IP address), deployment can be significantly more difficult. Also, if two tests are inadvertently run simultaneously, the system being tested may be under two separate test loads, possibly rendering the test results invalid. Avoidance of this may be difficult to detect whether one test is already being run at a particular time.